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VIDEO CODING METHOD OF EXPLOITING THE TEMPORAL REDUNDANCY BETWEEN SUCCESSIVE 
FRAMES 

5 The present invention relates to a video coding method of exploiting the temporal 
redundancy between successive frames in a video sequence. 

Efficiently encoding video or moving pictures relies heavily on exploiting the 
temporal redundancy between successive frames in a sequence. Based on the as- 

10 sumption that local motions are slow with respect to the temporal sampling pe- 
riod, several techniques have been proposed for efficiently removing this redun- 
dancy. The most successful and acclaimed method is block-based motion predic- 
tion, heavily used in nowadays standards such as MPEG4 and H.26L. Roughly 
speaking, these compression scheme predict a frame in the sequence based on the 

15 knowledge of previous frames. The current frame (or predicted) is cut into blocks 
of fixed size and the best matching block is searched in the reference frame. Dis- 
placement vectors are then encoded so that the decoder can reconstruct the predic- 
tion of the current frame from the previously decoded frame(s). As the block- 
based prediction is not accurate enough to encode perfectly the current frame, the 

20 error between the original and the predicted frame is encoded separately. This is in 
general referred to as texture coding or motion residual coding. The main draw- 
back of this method lies in the blocky nature of the prediction mechanism, which 
gives rise to very noticeable blocky artefacts at low bit rates. Moreover such a 
system, while well suited for wide translational motions, is unable to cope with 

25 locally complex movements or even global geometric transformations such as 
zoom or rotation. Finally, block based motion prediction is not able to follow 
m Ka ma& features of images since it is stuck in a fixed framework based on artificial 
image primitives (blocks). 
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This invention propose to solve the above-mentioned problems by introducing a 
new paradigm for dealing with spatio-temporal redundancy. 

The method according the invention is defined in claim 1. In the dependent claims 
various embodiments are proposed. 

The invention will be described with the help of accompanying representations. 

Figure 1: shows the progressive reconstruction of an I frame with 50, 100, 150 and 
200 atoms. 

Figure 2 : shows a flow-chart of the I-frames codec, where the atoms are used to 
transmit the frame. 

Figure 3: shows a flow-chart of the I-frames codec, where the atoms are estimated 
from the coded image both at the encoder and decode. 

Figure 4 : shows three successive schematic updates of basis functions inside a 
sequence of frames 

Figure 5 : shows a flow-chart of the P-frames coder, using atom based motion 
prediction. 

Figure 6 and 7: show the encoding of the Foreman sequence: I-frame, predicted 
frames #20, 50 and 99. 

Figure 8: represents PSNR along the encoded Foreman sequence as a function of 
the number of predicted frames. 
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The proposed method is composed of two main parts. In a first step, a geometric 
model of a reference frame is built. In the second step, the model is updated by 
deformations in order to match successive frames. 

A reference frame (or intra frame, I-frame) is first decomposed into a linear com- 
bination of basis functions (atoms) selected in a redundant, structured library (see . 
P. Vandergheynst and P. Frossard, Efficient image representation by anisotropic 
refinement in matching pursuit, in Proceedings of IEEE ICASSP, Salt Lake City 
UT, May 2001, vol. 3, the content of which is incorporated herein by reference) 

AT- 1 

/IS=0 

In this equation, I(x,y) is the Intensity of the I-frame represented as a function giv- 
ing the gray level value of the pixel at position (x,y). c n are weighting coefficients 
and g Yn (x, y) are the atoms involved in the decomposition. These atoms are image 
patches generated by a simple mathematical formula expressing the gray level 
value at pixel position (x,y). The formula is built by applying geometric transfor- 
mations to a function g(x,y) that we call a generating mother function. The pa- 
rameters of these transformations are concatenated in the vector of parameters y. 
Examples of possible transformations are translations, rotations or dilations. They 
act on the generating mother function by change of variable, for example: 



Translations: g b (x,y) = g(x-b It y-bi) 

Dilation: g Q ( x ,y) = a 1 g(x/a,y/a) 

Translations, anisotropic dilations and rotations: 
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_ cos^(x-b,)-sin3(y-b 2 ) 



a, 



S r (x,y) = g(x n> y n ), where 

_ sin^(x-b,) + cosS(y-b 2 ) 



y, 

and y=[a ls a2,bi,b2,&], are the parameters of this transformation 



a 2 



Generating mother functions are chosen almost arbitrarily. Their properties can be 
adapted to the specific application. A possible example is to select an oscillating 
function of the form: 



g(x,y) = (l-x 2 )exp^ 2 + y2 



The decomposition can for example be accomplished using a Matching Pursuit 
(see S. Mallat and Z. Zhang, Matching Pursuits with time-frequency dictionaries, 
IEEE Transactions on Signal Processing, 41(12):3397-3415, December 1993, the 
content of which is incorporated herein by reference). Matching Pursuit (MP) is a 
greedy algorithm that iteratively decomposes the image using the following 
scheme. First the atom that best matches the image is searched by maximizing the 
scalar product between the image and the dictionary (l|g r ) , and a residual image 
is computed: 

Then the same process is applied to the residual: 

M R -kX+R 2 > 



and iteratively: 
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R n ={R n |g, o )g ra +R n+l . 

Finally this yields a decomposition of the image in terms of a sum of atoms: 

5 

I W=Z(Rn|g rn )g, a (x) + R N ( X ). 
11=0 

The basis functions (atoms) are indexed by a string of parameters y n representing 
geometric transformations applied to a generating mother function g(x,y). This 
10 index can be seen as a point on a manifold. The set of geometric transformations 
is designed in such a way that the total collection of basis functions (atoms) is a 
dense subspace of L 2 (R 2 ) , i.e. any image can be exactly represented. 

This part of the method expresses the I-frame as a collection of atoms that can be 
15 seen as geometric features such as edges or parts of objects which are very notice- 
anie ov the numan eye. These basic primitives hereafter referred to as atoms, form 
a primal sketch of the image. The atoms are modelled and fully represented by the 
set of coefficients and parameters {c n , y n , n = 0,...,N-l} , where Cn is the coeffi- 
cient and y n is a vector of parameters. 

20 

^ -re are two ways to handle the coding of the I-frames. The first one is to esti- 
*%iixce the atoms of the original frame. The atoms modelling the I frame are then 
quantized, entropy coded and sent in the bitstream. The process of quantizing an 
iH^ti^i^rBesponds to the quantization of its coefficient and parameters. The atoms 
25 ai e msu stored in memory in order to be used for the prediction of the next frames. 
A flowchart of this procedure is shown in Figure 2. where the atoms are used to 
transmit the frame. Dotted lines represent the coefficients and parameters of the 
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atoms, the bold solid line represents a video frame and the light solid line repre- 
sents bitstreams. The flow-chart should be understood as follow: The input frame 
is decomposed into atom using the MP algorithm. The atoms are then quantized 
and entropy. They are also de-quantized and stored in memory for the prediction 
5 of the next frames. Those de-quantized atom parameters are used to reconstruct 
the image, with the help of the generating mother ftmction (which is known both 
by the encoder and decoder). Finally the difference between the original frame and 
the reconstructed one is computed and encoder using the frame coder. 

10 Note that the figure includes an optional step, that encodes the motion residuals 
(or texture), which is the difference between the original frame and the one recon- 
structed using the atoms. This encoding can be used to further increase the quality 
of the decoded image up to a lossless reconstruction. 

15 The second way of handling the I-frames is more conventional. The original frame 
is encoded and transmitted using any frame codec. Then the atoms are estimated 
from the reconstructed frame, both at the encoder and at the decoder. Finally those 
atoms are stored in memory for the prediction of future frames. The flowchart of 
this procedure is shown in Figure 3, where tfeK^ss are estimated from the coded 

20 image both at the encoder and decode. Dotted lises represent the coefficients and 
parameters of the atoms, the bold solid line represents a video frame and the light 
solid line represents bitstreams. The flow-chart should be understood as follow: 
The input frame is first encoder with the frame coder and send to the decoder in a 
bitstream. It is also decoded, in order to get ihe exact same frame that will be 

25 . available at the decoder. This frame is de@&xnp»sed into atoms by the MP algo- 
rithm. Finally those atoms are stored in memory to be used for the prediction of 
the next frames. 
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The second step of the method consists in updating the image model (the set of all 
atoms) in order to take into account the geometric deformations that have occurred 
between the reference and the current frame. Clearly, since the model is based on 
geometric transformations, updating its atoms allows for adapting to smooth local 
distortions (translations, rotations, scales are common examples). In order to com- 
pute this update, we assume that the deformed model is close enough to the refer- 
ence model. We thus have to search for new atoms parameters in the proximity of 
the previous solution. This is performed by means of a local optimization proce- 
dure trying to minimize the mean square error between the updated model and the 
current frame (Figure 4, where three successive schematic updates of basis func- 
tions (atoms) inside a sequence of frames are represented). The updated atom pa- 
rameters are then the solution of: 

ArgMinj Z c ng, D -it 

{cu.r.} Ul n 

where the optimization method for frame I t at time t is initialized with the atom 
parameters corresponding to the solution at timet -1 or to the reference frame (the 
I frame) in order to avoid error propagation. This problem is a non-convex, non- 
linear, differentiable optimization problem (see Dimitri P. Bertsekas (1999) 
Nonlinear Programming: 2nd Edition. Athena Scientific 
http://www.athenasc.com/nonlinbook.html, , the content of which is incorporated 
Jj^ein by reference), which can be solved using various algorithms such as quasi- 
Newton methods, combined with line-search or trust-region globalization tech- 
niques (see Conn A., Gould N. & Toint Ph. (2000) Trust Region Methods. SIAM. 
hap://www.fundp.ac.be/~phtoint/pht/trbook.html, the content of which is incorpo- 
rated herein by reference ), in order to identify a local optimum. 
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The difference between the original atom parameters and the updated ones is then 
computed and sent together with updated atom coefficients. Quantization and en- 
tropy coding can then be performed (P. Frossard, P. Vandergheynst and R.M, Fi- 
gueras y Ventura, Redundancy driven a posteriori matching pursuit quantization. 
5 ITS Technical Report 2002, the content of which is incorporated herein by refer- 
ence ) and a bit stream generated. This procedure is shown in detail in the flow- 
chart of Figure 5. representing a flow-chart of the P-frames coder, using atom 
based motion prediction. Dotted lines represent the coefficients and parameters of 
the atoms, the bold solid line represents a video frame and the light solid line 

10 represents bitstreams. The flow-chart should be understood as follow: The input 
frame are passed to the parameter estimation, which will modify the atom parame- 
ters stored in memory in order for them to describe correctly the input. The differ- 
ence between the new atom parameters and the one stored in memory is computed 
and the result is quantized and entropy coded. Those quantized difference are also 

15 de-quantized and added to the atom parameters previously stored in memory. This 
allows the reconstruction of the same atoms as the one available at the decoder. 
Those reconstructed atom parameters are then stored in memory, replacing the 
ones of the previous frame. They are also used to reconstruct the current frame 
using the generating mother function available at the encoder and decoder. The 

20 difference between this reconstruction and the original input is computed and en- 
coded using the frame coder. The number of updated atoms and their quantization 
can be fixed but can also be chosen in adaptive manner through rate and distortion 
constraints by means of a rate controller. For each motion predicted frame, the 
motion residuals (or texture) can be computed and encoded using any still image 

25 codec. It would then be sent in the bitstream in order to generate a scalable stream 
achieving lossy to lossless compression. 

Typical results of the motion prediction, wuh a coding cost of 1 70 Kbps in aver- 
age are shown in Figure 6, where predicted frames 20, 50 and 99 are represented. 
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It can be seen that the motion prediction stays accurate even after 100 frames, 
even in the absence of encoding of the motion residual. This is the major advan- 
tage of this technique compared to block based compensation. In order to show 
the advantage of the new prediction technique, we have done the same experiment 
with typical block matching compensation. The same sequence was encoded using 
adaptive block compensation with block sizes of 4x4 to 32x32. The encoder 
automatically selected the best block size according to Rate-Distortion optimisa- 
tion. By varying the block size it is possible to control the size of the compressed 
bitstream in order to match the result of the atom based motion prediction. Like in 
the case of the atom based prediction, the motion residual where not coded. The 
Motion prediction results of the block matching are shown in Figure 7. 

Objective comparison can be done using the PSNR measure. This measure is 
computed using the squared difference between the original and the reconstructed 



frame, i.e. = -lOlog 



255 2 



\ x *y J 



, where I(x,y) is the original 



frame and I r (x t y) is the reconstructed frame. The PSNR comparisons of the two 
prediction methods, for a bitstream of average size of 170 Kbps, are shown in 
Figure 8. The performance of the current atom based motion prediction is particu- 
larly good taken into consideration, that the states of the art block based motion 
prediction was used. Moreover, it can be seen that in the long term (more than 50 
frames) the prediction of the atom based method is more constant. In typical block 
matching applications, the block size is limited to 8x8, which causes poorer per- 
formances for the Block Matching. 



The number of predicted frames can be fixed but can also be chosen in adaptive 
manner through rate and distortion constraints by means of a rate controller. For 
instance, when abrupt changes occur (shots between scenes), the model is not an 
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accurate base anymore and the reference I-frame can simply be refreshed in order 
to compute a new model. Such a refresh mechanism can be monitored by tracking 
frame-to-frame distortion, which should stay rather constant for smooth updates of 
the initial model (Figure 8). 
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CLAIMS 

1. Video coding method of exploiting the temporal redundancy between 
successive frames in a video sequence characterized in that a reference 
frame, called I-frame, is first approximated by a collection of geometric 
features, called atoms, and that the following predicted frames called, 
P-frames, are approximated by the geometric transformations of the geo- 
metric features (atoms) describing the previous frame. 

2. Video coding method according to claim 1, characterized in that the I- 
frame is approximated by a linear combination of N atoms g (x,y): 

r (x,y) = ^c n g rm (x 9 y) 9 selected in a redundant, structured library and in- 

dexed by a string of parameters y n representing the geometric transforma- 
tions applied to the generating mother function g(x,y) and the c n are 
weighting coefficients. 

. Vkieo coding method according to claim 2, characterized in that the at- 
oms occurring in the decomposition are chosen using the Matching Pursuit 
algorithm. 

*** video coding method according to one of the claims 1 to 3, characterized 
» that the parameters and coefficients of the atoms are quantized and en- 
j# tropy coded. 

5. Video coding method according the claims 4, characterized in that the 
^iantization of the parameters and the coefficients can vary across time, 
and that the variation is controlled by a rale control unit. 
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6. Video coding method according to one of the claims 1 to 5, characterized 
in that the system is used as a motion prediction, and that the differences 
between the original frames and the ones reconstructed using the atoms, 
called the residual images, are encoded using another frame based codec. 

7. Video coding method according to one of the claims 1 to 6, characterized 
in that the geometric features (atoms) of the I-frame are computed from 
the quantized frames at the encoder and decoder and are not transmitted. 

8. Video coding method according to one of the claims 1 to 7, characterized 
in that the geometric features (atoms) are re-computed after each quan- 
tized frame at the encoder and decoder and replace the previous prediction. 

9. Video coding method according to one of the claims 1 to 8, characterized 
in that the geometric transformations used to build the library are com- 
posed of translations, anisotropic dilations and rotations, applied to a gen- 
erating mother function g(x,y) by m^sns of the following change of vari- 
ables: 

g r (*> y) = g(x n >y„), where 
_ cosa(x~b 1 )-sin&(y~b 2 ) 



10. Video coding method according to one -of the claims 1 to 9, characterized 
in that the generating mother function is of the following form: 



a, 



sin $(x -b, ) + CQ&8/y 
a 2 



u \ 
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